Infrared and visible light dual sensor imaging system

ABSTRACT

A dual sensor imaging system is described for visible and infrared light. One example includes a first image sensor to detect the luminance of a scene, a second image sensor to detect the visible light chrominance of the scene and to detect an infrared image of the scene, and an image processor to receive the luminance from the first image sensor and the chrominance from the second sensor to generate a visible light image of the scene, the image processor to further receive the infrared image from the second image sensor and to extract the infrared image from the visible light chrominance of the scene.

FIELD

The present disclosure relates to near infrared and visible light imageand video capture and, in particular, to a dual sensor camera system andmethod.

BACKGROUND

The steady decline in size and cost for digital camera modules resultsin ever more uses and installations of camera modules. Notebookcomputers, tablets, smart phones, and even some desktop monitors includea microphone and a camera near the display for use with videoconferencing. Some designs have multiple microphones to help with audionoise reduction. Some designs have multiple cameras to provide depthsensing, and other effects. The currently most common camera modules usea CMOS (Complementary Metal Oxide Semiconductor) sensor or photodetectorarray which is sensitive to all colors of visible light and also to nearinfrared (NIR) light. As a result any inexpensive camera module may beconverted to an infrared camera by removing the color filters andreplacing the NIR blocking filter with an appropriate NIR pass filter.

Notebook computers are now providing an NIR camera together with theprimary video camera. The NIR camera is provided for face log in but maybe used for any other function with the appropriate software. As analternative, a single camera may be configured to incorporate both NIRand RGB (Red Green Blue) visible light sensitivity. This allows bothfunctions to be performed with a single camera module, but the result islarger than the standard primary camera with worse optical performanceand more noise.

For clamshell and notebook form factors, a thin display (reducedZ-height) allows for a lighter, easier to use, and more attractivedevice. For monitors and displays, OLED (Organic Light Emitting Diode)and advanced LCD (Liquid Crystal Display) technologies allow for eventhinner desktop monitors. As these and portable devices compete formarket share, two areas of competition are to reduce the thickness ofthe device and to reduce the width of the bezel. Thinner or flatterdevices are perceived to be more stylish and in some cases fit moreeasily in slim pockets. Smaller bezels also allow more of the device tobe used for the display which increases the ease in reading the display.In some cases, it allows a device with a small screen to be made evensmaller for higher portability.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments are illustrated by way of example, and not by way oflimitation, in the figures of the accompanying drawings in which likereference numerals refer to similar elements.

FIG. 1 is a block diagram of dual camera composite image system with IRimaging according to an embodiment.

FIG. 2 is a graph of a filter transmittance characteristic for thecolor/IR sensor according to an embodiment.

FIG. 3 is a diagram of a color filter layout for the color/IR sensoraccording to an embodiment.

FIG. 4 is an isometric diagram of a video conferencing node according toan embodiment.

FIG. 5 is an isometric diagram of an alternative video conferencing nodeaccording to an embodiment.

FIG. 6 is a block diagram of a computing device incorporating IR cameraenhancements according to an embodiment.

DETAILED DESCRIPTION

A high quality, high definition camera provides a real benefit in videoconferencing and any other video or photography application. A largercamera sensor receives more light and so it works better with less noisewhen the light is low. On the other hand, the higher the quality, thehigher the resolution, and the larger the sensor, the larger theresulting camera module. As displays become very thin, a camera modulewith a CMOS sensor and an imaging lens may be thicker than the display.A high quality camera may also require a large sensor and also a largeaperture, autofocus lens system. Many otherwise thin smart phone deviceshave a camera hump on the back. By adding more cameras, the hump is madelarger.

As described herein, a pair of cameras or sensors may be combined toserve two or more functions. This results in a smaller lower cost systemthan if one camera performed both RGB and IR tasks or if two cameraseach performed a different task separately. By optimizing and combiningthe outputs of the two sensors for both tasks, the cost is lowered, theZ-height is reduced, and the performance is improved.

As described herein, the functions of a camera pair are configured toprovide an aggregate performance both for RGB and for NIR usages. Afirst camera of the pair is a monochrome sensor that is configured toprovide high luminance resolution with good sensitivity and low noise. Asecond camera of the pair is configured to provide lower resolutioncolor and NIR data. A monochrome camera can be combined with a colorcamera to deliver twice the effective resolution of a similar singlecamera. The improved resolution is provided together with improvementsin sensitivity and noise. As described herein, the dual camera system isoptimized for both RGB and NIR usages.

As an example, one camera is optimized to deliver high resolution forthe luminance portion of the RGB visible color applications while thesecond camera is optimized to deliver color and NIR data at a lowerresolution. The combination provides a lower Z-height. This is becausetwo 720P cameras are thinner than one 1080P camera. A similar ruleapplies to any resolution comparison. The combination provides lowercost because two 720P cameras cost less than two 1080P cameras or even acombination of a 1080P camera with a VGA (Video Graphics Adapter 480P)NIR camera. In addition, a monochrome sensor as described hereinprovides high signal quality even with smaller pixels. The combinationalso provides improved RGB performance compared to an RGB sensor. Thisis because a monochrome sensor has better sensitivity and dynamic rangethan a color sensor. The relatively lower resolution of the colorinformation does not affect the perceived RGB image quality as the humaneye is less sensitive to noise and resolution in the color space than inthe luminance space.

In a conventional system, such as a video conferencing system in anotebook computer or desktop display, a primary high quality 1080P RGBvideo camera may be used for a videoconference. This camera is combinedwith a VGA resolution NIR stream for face recognition or other NIR uses.The VGA NIR camera uses the same field of view as the high qualitycamera. The first primary RGB sensor provides luminance information andchrominance information for the scene. The primary camera may be large,expensive, and thick. As 4K and higher resolutions are developed, thesensor and the necessary optical system become larger and moreexpensive.

When a single camera is used to produce both RGB and NIR images, theremay be high color noise caused by the optical and electrical crosstalk.The crosstalk occurs because the color filters of the image sensor arenot selective in the NIR spectrum. The red, green, and blue filters alsopass NIR so the NIR content must be removed from each color channel.This cannot be accurately done because, the amount of NIR is differentin each color channel and the NIR photons tend to pass through onephotodetector into the next, blurring the distinctions. As a result, NIRextraction algorithms tend to add noise, which appears as differenttextures overlayed on the actual scene. A separate luminance sensor,which rejects all NIR light avoids this problem of subtracting the NIR.

For an RGB sensor, the sensor generates a brightness level for red,green, or blue light at each pixel or location on the sensor or atnearby locations. An equivalent way to characterize an image from thesensor is to measure the overall brightness, referred to as luminance,of a pixel and then combine that with the color of the pixel, referredto as chrominance, e.g. YCrCb. Human vision is particularly sensitive toluminance and to the brightness of the green part of a scene. Thespectral response of the green channel matches human perception ofluminance much more closely than the red or green channels, and so thegreen channel contains the majority of the luminance information fromthe camera. As described herein, the RGB/IR camera replaces some portionof the green pixels with NIR pixels. This effectively reduces theeffective luminance resolution of the sensor, but the monochrome sensormore than compensates for this loss of information. Green helps tobetter distinguish skin tones and plants which are particularly relevantfor a conferencing system.

Almost all camera image sensors use the Bayer or RGGB pixel patternwhich optimizes the image sensor for a luminance chrominance system byusing more green pixels than red or blue. A Bayer pattern of pixels issometimes referred to as RGGB to indicate an extra green pixel for eachred and blue pixel. The chrominance information requires that therelative brightness of the red, green, and blue all be available. Inmany video systems, RGB data is not transmitted, but luminance andchrominance, e.g. YCrCb, with the chrominance information transmitted atrelatively lower resolution and higher compression. This allows imagesto be sent with less data. The loss of chrominance data is difficult forhuman vision to perceive.

FIG. 1 is a diagram of a sensor system that uses two cameras 102, 104.The first camera works with the second camera for high resolution video132. The second camera works alone for low resolution NIR imaging 134.In this example, two 720P cameras together provide high resolution videowith the same effective resolution as a single 1080P RGGB camera. 1080Phas about two million pixels or about 2MP and 720P has about 1MP so thatadding the pixels from two 720P sensors provides an approximation of thedesired 2MP of resolution. The resolution values can be scaled to suitany desired implementation. As an example, two 480P (VGA) cameras may beused to produce 720P video. Higher resolutions are also possible.

The first camera has an optical imaging lens 110 to capture and focuslight from the scene. For a typical notebook computer video conferencingapplication, this lens is a simple fixed focus, fixed aperture lens. Forother systems, the lens may be more complex and may be able to vary oneor more of focus distance, aperture, and focal length. This light isfiltered by an IR cut-off filter 112 and then passed to a monochromesensor 114. The IR cut-off filter blocks light in the IR range forexample, with wavelengths longer than 650 nm. The filter may also be setto allow only visible light (about 400-650 nm) or only some narrowerrange of light near green light (about 475-575 nm) to pass through tothe sensor. The monochrome sensor is typically a standard CMOSphotodetector array and receives all of the light in the visible band ateach pixel. Accordingly, it only records luminance with no colorinformation.

The second camera 104 has a similar optical imaging lens 120 to captureand focus light from the same scene. The captured light is passedthrough an optical filter 122 that passes visible light of all visiblecolors and a particular, selected NIR band. FIG. 2 is an opticaltransmittance diagram for a suitable optical filter. The transmittancehas a visible light peak 202 from about 375 nm to 650 nm. Light with alonger wavelength than 650 is cut off by a transmittance floor 204, 208.However, there is a narrow band transmittance peak matching the NIRillumination wavelength, as an example at about 850 nm, to allow anarrow NIR band to pass through. The narrow NIR passband 206 has a verylow transmittance on either side 204, 208.

The light that passes through the filter is captured by the camerasensor 124 that is configured to sense RGB and NIR (essentially blackpixels). In some examples, the NIR light is that light needed for a facelogin application. Any desired NIR band may be selected. An example ofthe camera sensor filter structure is shown in FIG. 3. The sensor 124 inthis example is very similar to a conventional Bayer pattern except thatevery other green pixel is replaced with a NIR pixel. Since a typicalCMOS sensor array is sensitive to visible and NIR light, the sensor maybe adapted simply by changing the filter array. In this example, becausethe main filter 122 allows some NIR to pass, the red, green, and bluepixels may have an additional NIR filter or coating to block NIR lightfrom the color filters. Alternatively, the NIR values from the NIRpixels may be digitally subtracted from the red, green, and blue valuesduring post processing in the image signal processor 106.

In FIG. 3, the first row has a color filter over each photodetector,photosite, or pixel. The first, third, fifth, and subsequent odd filterspass red light while the second, fourth, sixth, and subsequent evenfilters pass blue light to the corresponding photodetector. The secondrow has a similar alternating pattern with the odd filters passing NIRlight and the even filters passing the blue light. In this array, thereis an equal number of red, green, blue, and infrared pixels. The red,green and blue pixels provide rich chrominance information for thevisible light spectrum. In a Bayer pattern the infrared filters would begreen. The green pixel information would be combined to provide theluminance information. The specific pattern of FIG. 3 is provided as anexample, there are many other variations. One variation is for the firstrow to be red and infrared and the second row to be blue and green. In afurther variation, the second image sensor has more pixels but theinfrared pixels still add up to a VGA resolution so that a largerpercentage of the pixels are used for red, green, and blue.

In the RGB image sensor 124, the colored RGB and IR filters reduce theamount of light or number of photons falling on the photosite byeliminating all of the other colors. As a result, the number ofelectrons generated at the photosite is reduced. This reduces the signalto noise ratio (SNR) by reducing the signal amplitude. This is a problemwith any color image sensor that relies on color filters to distinguishbetween the different colors in the scene.

However, the monochrome image sensor 114 of the first camera 102 doesnot require any color filters. As a result, the amount of light at eachpixel is much greater and the signal will be much higher for a muchbetter SNR. Configuring the imager on luminance reduces the spectralbandwidth of the optics allowing a sharper image given the same sourcesof chromatic aberration as compared to a sensor spanning both visibleand NIR spectrums. For a typical YCrCb 1080P video using a Bayer patternRGGB sensor, the majority of the luminance information comes from thegreen pixels which form about half of the pixels. In the example of FIG.1, these pixels are delivered using the 720P monochrome sensor that hasthe same number of luminance sensing elements, i.e. green elements, as a1080P RGGB sensor and therefore offers similar spatial, luminanceresolution. The visible light chrominance values come from the RGBpixels of the color sensor 124.

As shown in FIG. 2, wavelengths between the visible and desired NIRspectrum are blocked out, reducing the interference from other sourcesof NIR illumination, such as sunlight. The dual cameras are tuned tomeet the aggregate usage needs. The monochrome sensor is tuned todeliver luminance resolution equivalent to that of a 1080P RGGB sensor.The monochrome sensor delivers accurate, low noise, high dynamic rangeluminance information with very high detail. By using the full visiblespectrum instead of only green pixels, a higher signal level iscaptured. The RGB NIR sensor is tuned to deliver 720P color informationto combine with the luminance information and VGA IR information ondemand.

Returning to FIG. 1, each camera 102 104 also includes image correctioncircuitry 116, 126. This circuitry may take a variety of differentforms, depending on the nature of the image sensor and the opticalsystem. In the present example, each image correction circuit includesDefective Pixel Correction (DPC), Black Level Correction (BLC), andscaling from the original resolution to the output resolution such as720P to 1080P. The results from the image correction circuitry are twocorrected images. The first is a monochrome image of the scene with highdynamic range. The second is a color image with equal amounts of red,green, blue, and infrared. In some embodiments, the image correctioncircuit 126 for the color sensor 124 may be configured to produce eithera red, green, blue image or an infrared image at any time but not at thesame time. This allows the two functions to be separated and maysimplify processing. Alternatively, the infrared image may be isolatedby an image signal processor (ISP) 106.

The image signal processor 106 or another processing resource may alsobe used to control illumination sources. In this example, the systemincludes a NIR LED (Light Emitting Diode) 170 to provide NIRillumination for NIR images, and a white LED 172 to provide white broadband illumination for visible light images. The LEDs may be operatedtogether for composite color and IR images. The LEDs may be operatedindependently to provide only the type of light that is desired for thetype of video or image. There may be more than one of each type of LEDto provide a different field or direction of illumination or a differentcolor of illumination. Other light sources may also be used such asXenon flash, OLED, or other sources. The ISP may also adjust thebrightness, number and duration of the LEDs to suit different ambientenvironments or applications.

The two images are passed to an image signal processor (ISP) 106. Thisprocessor may be a discrete processor dedicated to images, it may be agraphics processor with multiple functions, it may be a centralprocessing unit (CPU) for general processing, or it may be a combinationof these and other types of processors. In the example of FIG. 1, theISP receives the color image and processes it to generate both aninfrared image and a color image. For an infrared image, the receivedinfrared pixels are rectified and scaled in an infrared processingmodule 128. This provides a VGA (480P) NIR image 134. This image maythen be provided to a variety of different functions, such as facerecognition, special imaging, or machine vision.

The system of FIG. 1 includes main processing 108 in the form of a CPU,SoC (System on a Chip), communications controller or other processingresource. The processing includes a face recognition module 150 attachedto local face image storage 152. The face recognition module comparesreceived NIR face images to the stored face images to determine whetherthe face is recognized. The results are passed to a log in module 154.

For producing a visible light image or video, the color sensor 124pixels are processed in a color image processing module 138. In thismodule, the color pixels are converted from the Bayer pattern to an RGBpattern with a demosaic, and CTC (Cross Talk Compensation) and colorconversion. The RGB pattern is thereby converted to a CrCb pattern. Thisalso removes the infrared pixels. The resulting image data is thencombined with the luminance data in a rectification/combination module118.

To combine the luminance sensor 114 data with the color/IR sensor 124data, the pixels must be correlated. Since the image sensors view thesame scene from a different position, there is a disparity between wherethe same features of the scene are imaged on each sensor. The disparitychanges with distance from the cameras due to parallax. As a result theparallax will be compensated with each image or frame of a video. Insome embodiments, the parallax is determined once for a video frame andthen adjusted periodically in the sequence of frames or when there is achange of scene. After the disparity between the images is determined,the luminance data may be combined with the appropriate color data tobuild a complete YCrCb image 132. The image or video sequence may beconverted or rendered in any of a variety of different forms, such asany of the MPEG (Motion Picture Experts Group) or JPEG (JointPhotographic Experts Group) formats. The image or sequence of images maythen be used by a video conferencing or other application.

The system of FIG. 1 includes main processing 108 coupled to receive thevideo stream. The processing 108 includes a video conference module 160that is coupled to a communications module 162. The communication may bewired or wireless to connect the system to one or more other videoconference nodes or to a recording or viewing station. The processingsends the frame sequence together with any associated microphone audioto other nodes and receives video and audio from other conference nodesfor display on a local monitor and playback through local speakers.While the illustrated example suggests that the ISP 106 is a localresource, the system may send the image data as two separate streams,one from each sensor. The combining functions may then be performed atanother conferencing node or by a conferencing server.

FIG. 4 is an isometric diagram of a portable device suitable for usewith the two part composite camera system as described herein. Thisdevice is a notebook, convertible, or tablet computer 220 with attachedkeyboard. The device has a display section 224 with a display 226 and abezel 228 surrounding the display. The display section is attached to abase 222 with a keyboard and speakers 242. The bezel is used as alocation to mount a luminance or monochrome camera 230 and a color/NIRcamera 232. The images detected by these two cameras may be separated orcombined as described above. The bezel may also be used to house an NIRLED flash 234, a white flash or lamp 236 and one or more microphones238, 240. In this example the microphones are separated apart to providea spatial character to the received audio. More or fewer microphones maybe used depending on the desired cost and audio performance. The ISP,graphics processor, CPU and other components are typically housed in thebase 222 but may be housed in the display section, depending on theparticular implementation.

This computer may be used as a conferencing device in which remote audiois played back through the speakers 242 and remote video is presented onthe display 226. The computer receives local audio at the microphones238, 240 and local video at the two composite cameras 230, 232. Thewhite LED 236 may be used to illuminate the local user for the benefitof the remote viewer. The white LED may also be used as a flash forstill imagery. The NIR LED 234 may be used to provide illumination forthe NIR pixels of the color and NIR camera 232. In one usage example,the color and NIR camera and the NIR flash are used to photograph auser's face for face recognition. The recognized face may then be usedas a log in.

FIG. 5 shows a similar device as a portable tablet or smart phone. Asimilar approach may be used for a desktop monitor or a wall display.The tablet or monitor 250 includes a display 252 and a bezel 254. Thebezel is used to house the various audiovisual components of the device.In this example, the bottom part of the bezel below the display housestwo microphones 256 and the top of the bezel above the display houses aspeaker 258. This is a suitable configuration for a smart phone and mayalso be adapted for use with other types of devices. The bezel alsohouses two composite cameras 260, 262 stacked on over the other, an NIRLED 264 and white LED 266. The various processors and other componentsdiscussed above may be housed behind the display and bezel or in anotherconnected component.

The particular placement and number of the components shown may beadapted to suit different usage models. More and fewer microphones,speakers, and LEDs may be used to suit different implementations.Additional components, such as proximity sensors, rangefinders,additional cameras, and other components may also be added to the bezelor to other locations, depending on the particular implementation.

In the example devices of FIGS. 4 and 5 fixed focus, fixed aperturecamera modules 230, 232, 260, 262 may be used. For these devices theuser is normally at about the same distance from the display. For othertypes of devices for which the user may be very close or very far fromthe camera variable or automatic focus camera modules may be used. For afixed focus camera module, the NIR LED may be configured to provide afixed amount of illumination based on the assumed distance of the user'sface from the flash.

The video conferencing nodes of FIGS. 4 and 5 are provided as examplesbut different form factors such as a desktop workstation, a walldisplay, a conference room telephone, an all-in-one or convertiblecomputer, and a set-top box form factor may be used, among others. Theimage sensors may be located in a separate housing from the display andmay be disconnected from the display bezel, depending on the particularimplementation. In some implementations, the display may not have abezel. For such a display, the microphones, cameras, speakers, LEDs andother components may be mounted in other housing that may or may not beattached to the display.

In another embodiment, the cameras and microphones are mounted to aseparate housing to provide a remote video device that receives bothinfrared and visible light images in a compact enclosure. Such a remotevideo device may be used for surveillance, monitoring, environmentalstudies and other applications. A communications interface may thentransmit the captured infrared and visible light imagery to anotherlocation for recording and viewing.

FIG. 6 is a block diagram of a computing device 100 in accordance withone implementation. The computing device 100 houses a system board 2.The board 2 may include a number of components, including but notlimited to a processor 4 and at least one communication package 6. Thecommunication package is coupled to one or more antennas 16. Theprocessor 4 is physically and electrically coupled to the board 2.

Depending on its applications, computing device 100 may include othercomponents that may or may not be physically and electrically coupled tothe board 2. These other components include, but are not limited to,volatile memory (e.g., DRAM) 8, non-volatile memory (e.g., ROM) 9, flashmemory (not shown), a graphics processor 12, a digital signal processor(not shown), a crypto processor (not shown), a chipset 14, an antenna16, a display 18 such as a touchscreen display, a touchscreen controller20, a battery 22, an audio codec (not shown), a video codec (not shown),a power amplifier 24, a global positioning system (GPS) device 26, acompass 28, an accelerometer (not shown), a gyroscope (not shown), aspeaker 30, cameras 32, a microphone array 34, and a mass storage device(such as hard disk drive) 10, compact disk (CD) (not shown), digitalversatile disk (DVD) (not shown), and so forth). These components may beconnected to the system board 2, mounted to the system board, orcombined with any of the other components.

The communication package 6 enables wireless and/or wired communicationsfor the transfer of data to and from the computing device 100. The term“wireless” and its derivatives may be used to describe circuits,devices, systems, methods, techniques, communications channels, etc.,that may communicate data through the use of modulated electromagneticradiation through a non-solid medium. The term does not imply that theassociated devices do not contain any wires, although in someembodiments they might not. The communication package 6 may implementany of a number of wireless or wired standards or protocols, includingbut not limited to Wi-Fi (IEEE 802.11 family), WiMAX (IEEE 802.16family), IEEE 802.20, long term evolution (LTE), Ev-DO, HSPA+, HSDPA+,HSUPA+, EDGE, GSM, GPRS, CDMA, TDMA, DECT, Bluetooth, Ethernetderivatives thereof, as well as any other wireless and wired protocolsthat are designated as 3G, 4G, 5G, and beyond. The computing device 100may include a plurality of communication packages 6. For instance, afirst communication package 6 may be dedicated to shorter range wirelesscommunications such as Wi-Fi and Bluetooth and a second communicationpackage 6 may be dedicated to longer range wireless communications suchas GPS, EDGE, GPRS, CDMA, WiMAX, LTE, Ev-DO, and others.

The cameras 32 including any depth sensors or proximity sensor arecoupled to an optional image processor 36 to perform conversions,analysis, noise reduction, comparisons, depth or distance analysis,image understanding and other processes as described herein. Theprocessor 4 is coupled to the image processor to drive the process withinterrupts, set parameters, and control operations of image processorand the cameras. Image processing may instead be performed in theprocessor 4, the cameras 32 or in any other device.

In various implementations, the computing device 100 may be a laptop, anetbook, a notebook, an ultrabook, a smartphone, a tablet, a personaldigital assistant (PDA), an ultra mobile PC, a mobile phone, a desktopcomputer, a server, a set-top box, an entertainment control unit, adigital camera, a portable music player, or a digital video recorder.The computing device may be fixed, portable, or wearable. In furtherimplementations, the computing device 100 may be any other electronicdevice that processes data or records data for processing elsewhere.

Embodiments may be implemented using one or more memory chips,controllers, CPUs (Central Processing Unit), microchips or integratedcircuits interconnected using a motherboard, an application specificintegrated circuit (ASIC), and/or a field programmable gate array(FPGA).

References to “one embodiment”, “an embodiment”, “example embodiment”,“various embodiments”, etc., indicate that the embodiment(s) sodescribed may include particular features, structures, orcharacteristics, but not every embodiment necessarily includes theparticular features, structures, or characteristics. Further, someembodiments may have some, all, or none of the features described forother embodiments.

In the following description and claims, the term “coupled” along withits derivatives, may be used. “Coupled” is used to indicate that two ormore elements co-operate or interact with each other, but they may ormay not have intervening physical or electrical components between them.

As used in the claims, unless otherwise specified, the use of theordinal adjectives “first”, “second”, “third”, etc., to describe acommon element, merely indicate that different instances of likeelements are being referred to, and are not intended to imply that theelements so described must be in a given sequence, either temporally,spatially, in ranking, or in any other manner.

The drawings and the forgoing description give examples of embodiments.Those skilled in the art will appreciate that one or more of thedescribed elements may well be combined into a single functionalelement. Alternatively, certain elements may be split into multiplefunctional elements. Elements from one embodiment may be added toanother embodiment. For example, orders of processes described hereinmay be changed and are not limited to the manner described herein.Moreover, the actions of any flow diagram need not be implemented in theorder shown; nor do all of the acts necessarily need to be performed.Also, those acts that are not dependent on other acts may be performedin parallel with the other acts. The scope of embodiments is by no meanslimited by these specific examples. Numerous variations, whetherexplicitly given in the specification or not, such as differences instructure, dimension, and use of material, are possible. The scope ofembodiments is at least as broad as given by the following claims.

The following examples pertain to further embodiments. The variousfeatures of the different embodiments may be variously combined withsome features included and others excluded to suit a variety ofdifferent applications. Some embodiments pertain to an apparatus thatincludes a first image sensor to detect the luminance of a scene, asecond image sensor to detect the visible light chrominance of the sceneand to detect an infrared image of the scene, and an image processor toreceive the luminance from the first image sensor and the chrominancefrom the second sensor to generate a visible light image of the scene,the image processor to further receive the infrared image from thesecond image sensor and to extract the infrared image from the visiblelight chrominance of the scene.

In further embodiments the first image sensor and the second imagesensor have the same number of pixels.

In further embodiments the first image sensor measures visible light ofall colors at each pixel.

In further embodiments the first image sensor is a monochrome sensor.

Further embodiments include an infrared light cut off filter between thefirst image sensor and the scene.

In further embodiments the second image sensor has an equal number orred, green, blue, and infrared pixels for example rows of alternatingred and green and rows of alternating blue and infrared.

Further embodiments include a filter between the second image sensor andthe scene with a narrow infrared passband filter centered for example at850 nm.

In further embodiments the visible light image generated by the imageprocessor has one of an RGB, JPEG, or YCrCb format.

In further embodiments the image processor is further to determine adisparity between the luminance and the chrominance of the scene and tocompensate for the disparity before generating a visible light image ofthe scene.

Further embodiments include a memory to store infrared face recognitionimages and a processor to receive the infrared image and perform facerecognition using the received image.

Further embodiments include an infrared LED to illuminate the scenewhile the second image sensor detects the infrared image of the scene.

Some embodiments pertain to a video conferencing node that includes amicrophone to receive local audio, a speaker to render remote audio, adisplay to render remote video, a communications interface to send andreceive audio and video with other video conferencing nodes, a firstimage sensor to detect the luminance of a local scene, a second imagesensor to detect the visible light chrominance of the local scene and todetect an infrared image of the scene, and an image processor to receivethe luminance from the first image sensor and the chrominance from thesecond sensor to generate visible light video of the local scene, theimage processor to further receive the infrared image from the secondimage sensor and to extract the infrared image from the visible lightchrominance of the scene.

In further embodiments the first image sensor has a plurality ofcomplementary metal oxide semiconductor pixels and measures visiblelight of all colors at each pixel.

Further embodiments include a filter between the second image sensor andthe scene with a narrow infrared passband filter and wherein the secondimage sensor has an equal number or red, green, blue, and infraredpixels.

Further embodiments include a memory to store infrared face recognitionimages and a processor to receive the infrared image and perform facerecognition using the received image.

Further embodiments include an infrared LED to illuminate the scenewhile the second image sensor detects the infrared image of the scene.

Some embodiments pertain to a method that includes detecting theluminance of a scene with a first image sensor, detecting the visiblelight chrominance of the scene with a second image sensor, detecting aninfrared image of the scene with the second image sensor, receiving theluminance from the first image sensor and the chrominance from thesecond sensor and generating a visible light image of the scene, andreceiving the infrared image from the second image sensor and extractingthe infrared image from the visible light chrominance of the scene.

In further embodiments detecting the chrominance and the infraredcomprises filter light from the scene to remove infrared light exceptfor a narrow infrared passband and simultaneously receiving visible andinfrared light at the second image sensor at red, green, blue, andinfrared pixels of the second image sensor.

Further embodiments include determining a disparity between theluminance and the chrominance of the scene and to compensate for thedisparity before generating a visible light image of the scene.

Further embodiments include receiving the infrared image and performingface recognition using the received image by comparing the receivedinfrared image with stored infrared face recognition images.

What is claimed is:
 1. An apparatus comprising: a first image sensor todetect the luminance of a scene; a second image sensor to detect thevisible light chrominance of the scene and to detect an infrared imageof the scene; and an image processor to receive the luminance from thefirst image sensor and the chrominance from the second sensor togenerate a visible light image of the scene, the image processor tofurther receive the infrared image from the second image sensor and toextract the infrared image from the visible light chrominance of thescene.
 2. The apparatus of claim 1, wherein the first image sensor andthe second image sensor have the same number of pixels.
 3. The apparatusof claim 2, wherein the first image sensor measures visible light of allcolors at each pixel.
 4. The apparatus of claim 2, wherein the firstimage sensor is a monochrome sensor.
 5. The apparatus of claim 1,further comprising an infrared light cut off filter between the firstimage sensor and the scene.
 6. The apparatus of claim 2, wherein thesecond image sensor has an equal number or red, green, blue, andinfrared pixels for example rows of alternating red and green and rowsof alternating blue and infrared.
 7. The apparatus of claim 1, furthercomprising a filter between the second image sensor and the scene with anarrow infrared passband filter centered for example at 850 nm.
 8. Theapparatus of claim 1, wherein the visible light image generated by theimage processor has one of an RGB, JPEG, or YCrCb format.
 9. Theapparatus of claim 1, wherein the image processor is further todetermine a disparity between the luminance and the chrominance of thescene and to compensate for the disparity before generating a visiblelight image of the scene.
 10. The apparatus of claim 1, furthercomprising a memory to store infrared face recognition images and aprocessor to receive the infrared image and perform face recognitionusing the received image.
 11. The apparatus of claim 10, furthercomprising an infrared LED to illuminate the scene while the secondimage sensor detects the infrared image of the scene.
 12. A videoconferencing node comprising: a microphone to receive local audio; aspeaker to render remote audio; a display to render remote video; acommunications interface to send and receive audio and video with othervideo conferencing nodes; a first image sensor to detect the luminanceof a local scene; a second image sensor to detect the visible lightchrominance of the local scene and to detect an infrared image of thescene; and an image processor to receive the luminance from the firstimage sensor and the chrominance from the second sensor to generatevisible light video of the local scene, the image processor to furtherreceive the infrared image from the second image sensor and to extractthe infrared image from the visible light chrominance of the scene. 13.The video conferencing node of claim 12, wherein the first image sensorhas a plurality of complementary metal oxide semiconductor pixels andmeasures visible light of all colors at each pixel.
 14. The videoconferencing node of claim 12, further comprising a filter between thesecond image sensor and the scene with a narrow infrared passband filterand wherein the second image sensor has an equal number or red, green,blue, and infrared pixels.
 15. The video conferencing node of claim 12,further comprising a memory to store infrared face recognition imagesand a processor to receive the infrared image and perform facerecognition using the received image.
 16. The video conferencing node ofclaim 15, further comprising an infrared LED to illuminate the scenewhile the second image sensor detects the infrared image of the scene.17. A method comprising: detecting the luminance of a scene with a firstimage sensor; detecting the visible light chrominance of the scene witha second image sensor; detecting an infrared image of the scene with thesecond image sensor; receiving the luminance from the first image sensorand the chrominance from the second sensor and generating a visiblelight image of the scene; and receiving the infrared image from thesecond image sensor and extracting the infrared image from the visiblelight chrominance of the scene.
 18. The method of claim 17, whereindetecting the chrominance and the infrared comprises filter light fromthe scene to remove infrared light except for a narrow infrared passbandand simultaneously receiving visible and infrared light at the secondimage sensor at red, green, blue, and infrared pixels of the secondimage sensor.
 19. The method of claim 17, further comprising determininga disparity between the luminance and the chrominance of the scene andto compensate for the disparity before generating a visible light imageof the scene.
 20. The method of claim 17, further comprising receivingthe infrared image and performing face recognition using the receivedimage by comparing the received infrared image with stored infrared facerecognition images.